Feedback based Neural Networks

The most successful learning models in deep learning are currently based on the paradigm of successive learning of representations followed by a decision layer. This is mostly actualized by feedforward multilayer neural networks, such as ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach, in which the representation is formed in a iterative manner according to a feedback received from previous iteration’s outcome. We establish that a feedback based approach has several fundamental advantages over feedforward: it enables making early predictions at the query time, its output conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We also investigate several new feedback mechanisms (e.g. skip connections in time) and design choices (e.g. feedback length). We hope this study offers new perspectives for investigating the feedback paradigm, in quest for more natural and practical learning models.